Skip to content

GH-1491: Release the batch when a C Data stream import fails before taking it - #1492

Open
sezruby wants to merge 3 commits into
apache:mainfrom
sezruby:fix/stream-reader-release-on-import-failure
Open

sezruby wants to merge 3 commits into
apache:mainfrom
sezruby:fix/stream-reader-release-on-import-failure

Conversation

@sezruby

@sezruby sezruby commented Nov 16, 2026 •

Copy link
Copy Markdown

What's Changed

ArrowArrayStreamReader.loadNextBatch now releases the batch when the import fails before moving it out of the array: allocating the schema struct on the first batch, or allocating the struct the batch moves into. Before this change, the struct was closed without calling its release callback, and the producer's batch leaked. This is the stream counterpart of #1239, which covered failures while importing the buffers.

The change and the test were written with AI assistance (Claude Code). I reproduced the leak and verified the fix locally.

Closes #1491.

Generated-by: Claude Code

…fore taking it

ArrowArrayStreamReader.loadNextBatch now releases the batch when the import
fails before moving it out of the array, e.g. allocating the schema struct or
the struct the batch moves into. Before, the struct was closed without calling
its release callback, which leaked the producer's batch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

This comment has been minimized.

Comment on lines +109 to +115
@Test
public void streamImportOomDoesNotLeakExportedBatch() throws Exception {
// Reading the schema lazily, the first batch can fail allocating the schema struct.
streamImportOomDoesNotLeakExportedBatch(false);
// Reading the schema first, it can fail allocating the struct the batch moves into.
streamImportOomDoesNotLeakExportedBatch(true);
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should be able to do this with JUnit 5's test parameterization

@sezruby sezruby Nov 17, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in dc15243: it is now one @ParameterizedTest with @ValueSource(booleans = {false, true}).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment on lines +67 to +76
try {
Data.importIntoVectorSchemaRoot(allocator, array, getVectorSchemaRoot(), provider);
} catch (Throwable t) {
// If the import failed before moving the batch out of the array, e.g. allocating the
// schema, the batch is still in the array: release it.
if (array.snapshot().release != NULL) {
array.release();
}
throw t;
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR wraps the 4-argument call:

    try {
      Data.importIntoVectorSchemaRoot(allocator, array, getVectorSchemaRoot(), provider);
    } catch (Throwable t) {
      if (array.snapshot().release != NULL) {
        array.release();
      }
      throw t;
    }

The 4-argument Data.importIntoVectorSchemaRoot defaults to closeImportedStructs = true. Inside the 5-argument Data.importIntoVectorSchemaRoot:

  1. Line 487 calls importIntoVector(..., closeImportedStructs) which calls array.close().
  2. Line 488-492 then run StructVectorUnloader.getRecordBatch() and VectorLoader.load(recordBatch).

If anything throws during unloading or loading (for example, an OutOfMemoryException in BaseVariableWidthVector.setReaderAndWriterIndex when allocating an offset buffer, or an IllegalArgumentException in StructVectorUnloader / VectorLoader), execution enters the catch (Throwable t) block while array is already closed. Calling array.snapshot() then hits checkNotNull(data, "ArrowArray is already closed") in ArrowArray.memoryAddress, throwing NullPointerException and masking the original exception t.

I suggest to pass closeImportedStructs = false since the enclosing try (ArrowArray array = ArrowArray.allocateNew(allocation)) already owns closing array (this also removes the redundant double-close() on the happy path):

Suggested change
try {
Data.importIntoVectorSchemaRoot(allocator, array, getVectorSchemaRoot(), provider);
} catch (Throwable t) {
// If the import failed before moving the batch out of the array, e.g. allocating the
// schema, the batch is still in the array: release it.
if (array.snapshot().release != NULL) {
array.release();
}
throw t;
}
try {
Data.importIntoVectorSchemaRoot(
allocator, array, getVectorSchemaRoot(), provider, /* closeImportedStructs= */ false);
} catch (Throwable t) {
// If the import failed before moving the batch out of the array, e.g. allocating the
// schema, the batch is still in the array: release it.
if (array.snapshot().release != NULL) {
try {
array.release();
} catch (Throwable suppressed) {
t.addSuppressed(suppressed);
}
}
throw t;
}

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, good catch. Applied in 2d037af: the import now passes closeImportedStructs = false, so the array stays open for the failure handling and the enclosing try-with-resources closes it once. A failure releasing the batch is added to the original exception as suppressed.

I didn't add a test for this path because I couldn't find an input that fails after the import closes the array. ArrayImporter.importBuffers loads the buffers into the vectors before the array is closed, including allocating a missing validity bitmap. After that, unloading the struct and loading the root only retain the imported buffers. setReaderAndWriterIndex allocates only when the offset buffer holds fewer than length + 1 offsets, and the importer rejects such a buffer. So the allocator-limit sweep in the stream test doesn't reach this path.

The 4-argument importIntoVectorSchemaRoot closes the array before
unloading and loading the batch, so a failure there made the failure
handling read a closed array: the NullPointerException masked the
original exception. Import with closeImportedStructs = false and let the
enclosing try-with-resources close the array, and add a failure to
release the batch to the original exception as suppressed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@jbonofre jbonofre left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @sezruby!

The stream reader releases the fetched array when import fails before ownership transfers, and the regression test covers failures with both lazy and eager schema initialization.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug-fix PRs that fix a big.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

C Data stream import leaks the batch when the import fails before taking it from the array

3 participants